79 research outputs found
Welcome to the modern world of pronouns: identity-inclusive Natural Language Processing beyond gender
The world of pronouns is changing. From a closed class of words with few
members to a much more open set of terms to reflect identities. However,
Natural Language Processing (NLP) is barely reflecting this linguistic shift,
even though recent work outlined the harms of gender-exclusive language
technology. Particularly problematic is the current modeling 3rd person
pronouns, as it largely ignores various phenomena like neopronouns, i.e.,
pronoun sets that are novel and not (yet) widely established. This omission
contributes to the discrimination of marginalized and underrepresented groups,
e.g., non-binary individuals. However, other identity-expression phenomena
beyond gender are also ignored by current NLP technology. In this paper, we
provide an overview of 3rd person pronoun issues for NLP. Based on our
observations and ethical considerations, we define a series of desiderata for
modeling pronouns in language technology. We evaluate existing and novel
modeling approaches w.r.t. these desiderata qualitatively, and quantify the
impact of a more discrimination-free approach on established benchmark data
University of Mannheim @ CLSciSumm-17: Citation-Based Summarization of Scientific Articles Using Semantic Textual Similarity
The number of publications is rapidly growing and it is essential to enable fast access and analysis of relevant articles. In this paper, we describe a set of methods based on measuring semantic textual similarity, which we use to semantically analyze and summarize publications through other publications that cite them. We report the performance of our approach in the context of the third CL-SciSumm shared task and
show that our system performs favorably to competing systems in terms of produced summaries
Values, Ethics, Morals? On the Use of Moral Concepts in NLP Research
With language technology increasingly affecting individuals' lives, many
recent works have investigated the ethical aspects of NLP. Among other topics,
researchers focused on the notion of morality, investigating, for example,
which moral judgements language models make. However, there has been little to
no discussion of the terminology and the theories underpinning those efforts
and their implications. This lack is highly problematic, as it hides the works'
underlying assumptions and hinders a thorough and targeted scientific debate of
morality in NLP. In this work, we address this research gap by (a) providing an
overview of some important ethical concepts stemming from philosophy and (b)
systematically surveying the existing literature on moral NLP w.r.t. their
philosophical foundation, terminology, and data basis. For instance, we analyse
what ethical theory an approach is based on, how this decision is justified,
and what implications it entails. Our findings surveying 92 papers show that,
for instance, most papers neither provide a clear definition of the terms they
use nor adhere to definitions from philosophy. Finally, (c) we give three
recommendations for future research in the field. We hope our work will lead to
a more informed, careful, and sound discussion of morality in language
technology.Comment: to be published in EMNLP 2023 Finding
Language representations for computational argumentation
Argumentation is an essential feature and, arguably, one of the most exciting phenomena of natural language use. Accordingly, it has fascinated scholars and researchers in various fields, such as linguistics and philosophy, for long. Its computational analysis, falling under the notion of computational argumentation, is useful in a variety of domains of text for a range of applications. For instance, it can help to understand users’ stances in online discussion forums towards certain controversies, to provide targeted feedback to users for argumentative writing support, and to automatically summarize scientific publications. As in all natural language processing pipelines, the text we would like to analyze has to be introduced to computational argumentation models in the form of numeric features. Choosing such suitable semantic representations is considered a core challenge in natural language processing. In this context, research employing static and
contextualized pretrained text embedding models has recently shown to reach state-of-the-art performances for a range of natural language processing tasks. However, previous work has noted the specific difficulty of computational argumentation scenarios with language representations as one of the main bottlenecks and called for targeted research on the intersection of the two fields. Still, the efforts focusing on the interplay between computational argumentation and representation learning have been few and far apart.
This is despite (a) the fast-growing body of work in both computational argumentation and representation learning in general and (b) the fact that some of the open challenges
are well known in the natural language processing community.
In this thesis, we address this research gap and acknowledge the specific importance of research on the intersection of representation learning and computational argumentation.
To this end, we (1) identify a series of challenges driven by inherent characteristics of argumentation in natural language and (2) present new analyses, corpora, and methods to address and mitigate each of the identified issues. Concretely, we focus on five main
challenges pertaining to the current state-of-the-art in computational argumentation:
(C1) External knowledge: static and contextualized language representations encode distributional knowledge only. We propose two approaches to complement this knowledge with knowledge from external resources. First, we inject lexico-semantic knowledge through an additional prediction objective in the pretraining stage. In a second study, we demonstrate how to inject conceptual knowledge post hoc employing the adapter framework. We show the effectiveness of these approaches on general natural language understanding and argumentative reasoning tasks.
(C2) Domain knowledge: pretrained language representations are typically trained on big and general-domain corpora. We study the trade-off between employing such large and general-domain corpora versus smaller and domain-specific corpora for training static word embeddings which we evaluate in the analysis of scientific arguments.
(C3) Complementarity of knowledge across tasks: many computational argumentation tasks are interrelated but are typically studied in isolation. In two case studies, we show the effectiveness of sharing knowledge across tasks. First, based on a corpus of scientific texts, which we extend with a new annotation layer reflecting fine-grained argumentative structures, we show that coupling the argumentative analysis with other rhetorical analysis tasks leads to performance improvements for the higher-level tasks.
In the second case study, we focus on assessing the argumentative quality of texts. To this end, we present a new multi-domain corpus annotated with ratings reflecting different dimensions of argument quality. We then demonstrate the effectiveness of sharing knowledge across the different quality dimensions in multi-task learning setups.
(C4) Multilinguality: argumentation arguably exists in all cultures and languages around the globe. To foster inclusive computational argumentation technologies, we dissect the current state-of-the-art in zero-shot cross-lingual transfer. We show big drops in performance when it comes to resource-lean and typologically distant target languages. Based on this finding, we analyze the reasons for these losses and propose to move to inexpensive few-shot target-language transfer, leading to consistent performance improvements in higher-level semantic tasks, e.g., argumentative reasoning.
(C5) Ethical considerations: envisioned computational argumentation applications, e.g., systems for self-determined opinion formation, are highly sensitive. We first discuss which ethical aspects should be considered when representing natural language for computational argumentation tasks. Focusing on the issue of unfair stereotypical bias, we then conduct a multi-dimensional analysis of the amount of bias in monolingual and cross-lingual embedding spaces. In the next step, we devise a general framework for implicit and explicit bias evaluation and debiasing. Employing intrinsic bias measures and benchmarks reflecting the semantic quality of the embeddings, we demonstrate the effectiveness of new debiasing methods, which we propose. Finally, we complement this analysis by testing the original as well as the debiased language representations for stereotypically unfair bias in argumentative inferences.
We hope that our contributions in language representations for computational argumentation fuel more research on the intersection of the two fields and contribute to fair, efficient, and effective natural language processing technologies
Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers
Following the major success of neural language models (LMs) such as BERT or
GPT-2 on a variety of language understanding tasks, recent work focused on
injecting (structured) knowledge from external resources into these models.
While on the one hand, joint pretraining (i.e., training from scratch, adding
objectives based on external knowledge to the primary LM objective) may be
prohibitively computationally expensive, post-hoc fine-tuning on external
knowledge, on the other hand, may lead to the catastrophic forgetting of
distributional knowledge. In this work, we investigate models for complementing
the distributional knowledge of BERT with conceptual knowledge from ConceptNet
and its corresponding Open Mind Common Sense (OMCS) corpus, respectively, using
adapter training. While overall results on the GLUE benchmark paint an
inconclusive picture, a deeper analysis reveals that our adapter-based models
substantially outperform BERT (up to 15-20 performance points) on inference
tasks that require the type of conceptual knowledge explicitly present in
ConceptNet and OMCS
How (Not) to Use Sociodemographic Information for Subjective NLP Tasks
Annotators' sociodemographic backgrounds (i.e., the individual compositions
of their gender, age, educational background, etc.) have a strong impact on
their decisions when working on subjective NLP tasks, such as hate speech
detection. Often, heterogeneous backgrounds result in high disagreements. To
model this variation, recent work has explored sociodemographic prompting, a
technique, which steers the output of prompt-based models towards answers that
humans with specific sociodemographic profiles would give. However, the
available NLP literature disagrees on the efficacy of this technique -- it
remains unclear, for which tasks and scenarios it can help and evaluations are
limited to specific tasks only. We address this research gap by presenting the
largest and most comprehensive study of sociodemographic prompting today.
Concretely, we evaluate several prompt formulations across seven datasets and
six instruction-tuned model families. We find that (1) while sociodemographic
prompting can be beneficial for improving zero-shot learning in subjective NLP
tasks, (2) its outcomes largely vary for different model types, sizes, and
datasets, (3) are subject to large variance with regards to prompt
formulations. Thus, sociodemographic prompting is not a reliable proxy for
traditional data annotation with a sociodemographically heterogeneous group of
annotators. Instead, we propose (4) to use it for identifying ambiguous
instances resulting in more informed annotation efforts
- …